Sample code to bypass twitcher to debug performance raven process timeout #5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Suspecting Twitcher is buffering the request and/or the response which makes Raven appears to take longer to respond to client request. This is meant as a temporary testing setup only, NOT to be enabled permanently on prod.
Left the code commented out so we have a sample how to do this as documentation.
…sometime 4 min was not enough Have to increase it here since this is an Ouranos prod environment specific problem therefore not appropriate to increase in the generic birdhouse-deploy repo.
Could not set it here due to this error: ``` + touch /etc/cron.d/canarieapi-cron + exec /bin/sh -c env >> /etc/environment && cron && nginx && gunicorn -b 0.0.0.0:2000 --workers 1 --log-level=DEBUG --timeout 30 -k geven t wsgi 2021/02/08 16:32:23 [warn] 10#10: duplicate MIME type "text/html" in /etc/nginx/conf.d/all-services.include:47 nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/conf.d/all-services.include:47 2021/02/08 16:32:23 [emerg] 10#10: "proxy_read_timeout" directive is duplicate in /etc/nginx/conf.extra.d/proxy/increase-proxy_read_timeout.conf:1 nginx: [emerg] "proxy_read_timeout" directive is duplicate in /etc/nginx/conf.extra.d/proxy/increase-proxy_read_timeout.conf:1 ```
tlvu
added a commit
to bird-house/birdhouse-deploy
that referenced
this pull request
Feb 10, 2021
…uld-be-configurable proxy: proxy_read_timeout config should be configurable We have a performance problem with the production deployment at Ouranos so we need a longer timeout. Being a Ouranos specific need, it should not be hardcoded as in previous PR #122. The previous increase was sometime not enough ! The value is now configurable via `env.local` as most other customizations. Documentation updated. Timeout in Prod: ``` WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/raven/wps FINCH_WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/finch/wps FLYINGPIGEON_WPS _URL=https://pavics.ouranos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Running_HMETS_with_CANOPEX_datas et.ipynb --sanitize-with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints HTTPError: 504 Server Error: Gateway Time-out for url: https://pavics.ouranos.ca/twitcher/ows/proxy/raven/wps ===================================================== 11 failed, 4 passed, 1 warning in 249.80s (0:04:09) =========================================== ``` Pass easily on my test VM with very modest hardware (10G ram, 2 cpu): ``` WPS_URL=https://lvupavicsmaster.ouranos.ca/twitcher/ows/proxy/raven/wps FINCH_WPS_URL=https://lvupavicsmaster.ouranos.ca/twitcher/ows/proxy/finch/wp s FLYINGPIGEON_WPS_URL=https://lvupavicsmaster.ouranos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Runni ng_HMETS_with_CANOPEX_dataset.ipynb --sanitize-with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints =========================================================== 15 passed, 1 warning in 33.84s =========================================================== ``` Pass against Medus: ``` WPS_URL=https://medus.ouranos.ca/twitcher/ows/proxy/raven/wps FINCH_WPS_URL=https://medus.ouranos.ca/twitcher/ows/proxy/finch/wps FLYINGPIGEON_WPS_URL=https://medus.ouranos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb --sanitize-with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints ============================================== 15 passed, 1 warning in 42.44s ======================================================= ``` Pass against `hirondelle.crim.ca`: ``` WPS_URL=https://hirondelle.crim.ca/twitcher/ows/proxy/raven/wps FINCH_WPS_URL=https://hirondelle.crim.ca/twitcher/ows/proxy/finch/wps FLYINGPIGEON_WPS_URL=https://hirondelle.crim.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb --sanitize-with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints =============================================== 15 passed, 1 warning in 35.61s =============================================== ``` For comparison, a run on Prod without Twitcher (PR bird-house/birdhouse-deploy-ouranos#5): ``` WPS_URL=https://pavics.ouranos.ca/raven/wps FINCH_WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/finch/wps FLYINGPIGEON_WPS_URL=https://pavics .ouranos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb --sanitize -with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints HTTPError: 504 Server Error: Gateway Time-out for url: https://pavics.ouranos.ca/raven/wps ================================================ 11 failed, 4 passed, 1 warning in 248.99s (0:04:08) ================================================= ``` A run on Prod without Twitcher and Nginx (direct hit Raven): ``` WPS_URL=http://pavics.ouranos.ca:8096/ FINCH_WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/finch/wps FLYINGPIGEON_WPS_URL=https://pavics.oura nos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb --sanitize-with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints ===================================================== 15 passed, 1 warning in 218.46s (0:03:38) ======================================================
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Just commented out sample code. Keep as a form of documentation how each organization can customize PAVICS deployment.
Used in bird-house/birdhouse-deploy#123 for testing.